MM Eval tests #1887
MM Eval tests #1887
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1887
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| ] | ||
|
|
||
|
|
||
| def dummy_vision_model(): |
There was a problem hiding this comment.
we have this in tests/torchtune/modules/test_common_utils.py, opportunity to unify?
There was a problem hiding this comment.
It made the most sense to me to define the fixture where it was being used - do you have strong opinions here?
| @@ -170,6 +170,7 @@ def llama3_2_vision_decoder( | |||
| by :func:`~torchtune.modules.KVCache`. | |||
| encoder_max_seq_len (int): maximum sequence length the encoder will be run with, as used | |||
| by :func:`~torchtune.modules.KVCache`. | |||
| rope_base (int): base for the rotary positional embeddings. Default: 500_000 | |||
There was a problem hiding this comment.
How did our linter not pick this up...
|
holy moly ============================= slowest 20 durations =============================
376.79s call tests/recipes/test_eleuther_eval.py::TestEleutherEval::test_meta_eval_vision
342.01s call tests/recipes/test_eleuther_eval.py::TestEleutherEval::test_hf_eval_visionI fix |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1887 +/- ##
===========================================
- Coverage 68.40% 24.78% -43.62%
===========================================
Files 311 318 +7
Lines 16973 17649 +676
===========================================
- Hits 11610 4375 -7235
- Misses 5363 13274 +7911 ☔ View full report in Codecov by Sentry. |
|
I DONT understand why this is failing, it's passing locally |
|
im so done |
Context
What is the purpose of this PR? Is it to
Please link to any issues this PR addresses.
Adds some tests for the eval recipe using dummy Llama3.2 Vision models.
Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example